Skip to main content

Incident Report: RUB/USD Self-Funded Feed Heartbeat Exceeded on Arbitrum

Date: 2023-12-16
Time: 12:25 (GMT+3)
Duration: 100.27 hours (as reported)

Description​

The RUB/USD self-funded feed on Arbitrum exceeded the 24-hour heartbeat interval, indicating that the data feed was not updated within the expected time frame. This was initially observed on the Centurion dashboard.

Root Cause​

The main cause was identified as Centurion pointing to the old Api3ServerV1 contract for RUB/USD data. Additionally, there was confusion about whether RUB/USD had been deployed to the new contract for Arbitrum. Misalignment between the Centurion configuration and the latest deployed contracts led to outdated data being read.

Impact​

Inaccurate or outdated information being displayed, potentially affecting decisions made based on this data.

Timeline​

  • 12:25 - Abdel reported the exceeded heartbeat interval for RUB/USD on Arbitrum.
  • 13:47 - Bedirhan verified that there is no problems on Nodary side. Suggested it may related to Centurion.
  • 02:16(4 days later) - Warren confirmed that Centurion does not reads the data from the new contract, leading to the decision to purge the PRICES table from Centurion and rebuild from the new contract.

Lessons Learned​

Regular verification of data sources and the proper alignment of monitoring tools with the current operational contracts are essential. Prompt and clear communication between team members can lead to quicker identification and resolution of data discrepancies.

Actions Taken​

  1. Initial identification of the heartbeat issue and documentation.
  2. Investigation by multiple team members into the status and health of the feed.
  3. Verification of the correct contract addresses and data feed deployment status.
  4. Decision to purge and rebuild the PRICES table in Centurion to align with the new contract data.

Incident Reviewer(s)​

  • Abdel (Reported the initial issue and followed up)
  • Ali (Initial assessment and request for check)
  • Bedirhan (Checked the issue and provided insights on the cause)
  • Warren (Conducted detailed analysis and implemented the fix)
Last updated on by Warren Anderson